Flights Delay prediction at New York.

Delay Prediction of Planes at New York.

Content

1. Problem Statement and Understanding.
2. Reading and binding the test data.
3. Data Cleaning.
4. Feature Engineering.
5. Checking and handling unbalanced data.
6. Checking for multicollinearity.
7. Model Building 7.1) classification 7.1.1) Logistic Regression. 7.1.2) KNN 7.1.3) Decision Tree. 7.1.4) Random Forest. 7.1.5) SVM. 7.1.6) XGboost.
8. Model selection for deployment. 9)conclusion.

Problem Statement

Understand the key factors through visualization that cause the delay Predict the possibility of delay in the future.

Understanding

The given datasets had the flight delay details of 2 Airports in the US namely JFK(NewYork) and DFW(Dallas).Identify the features which cause delays & use ML to predict future delays.

libraries used.

#used for reading and subsetting the data
library(data.table)
library(purrr)
#used for converting the format of the date column 
library(lubridate)
#used for plotting graphs
library(ggplot2)
library(plotly)
#used for smote function to do under sampling and over sampling
library(DMwR)
#The library for cross validation
library(caret)
library(mlbench)
#decision tree model building packages 
library(rpart)
#decision tree plotting packages 
library(RColorBrewer)
library(rattle)
#The package for random forest function
library(MASS)
library(randomForest)
#packages for xgboost
library(xgboost)
library(Matrix)
library(readr)
library(car)
#For one hot encode function
library(mltools)
#library for ROC curve
library(ROCR)
library(car)
#library for summarise function 
library(dplyr)
#This Library is used for reading Excel file.
library(readxl)
#For calculating the Geo distance.
library(geosphere)

Reading the csv files of all the airlines who ’s origin airport is NewYork.

#data<-fread("Detailed_Statistics_Departures.csv",skip = 7,stringsAsFactors = TRUE)
Total_data<- 
   list.files(pattern = "*.csv") %>% 
    map_df(~fread(.,skip = 7,stringsAsFactors = TRUE))

Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Discarded single-line footer: <<SOURCE: Bureau of Transportation Statistics>>Unequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorUnequal factor levels: coercing to characterbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vectorbinding character and factor vector, coercing into character vector

Data Cleaning

Now checking for the NA values for both train and test data sets.

#Checking the NA 'S at entire data set of train and test
cat("The Number of NA 'S in the data: ", sum(Total_data=="na"))

The Number of NA 'S in the data:  0

#checking for the empty sells in the table
cat("\nThe Number of blank cell 'S in the data: ", sum(Total_data==""))

The Number of blank cell 'S in the data:  200

As we see above output their are 200 empty spaces in the training data, so now we are going to see the empty rows for finding the reason that why they are empty

Now seing in which columns empty spaces are present.

apply(Total_data,2,function(x)sum(x==""))

                            Carrier Code                        Date (MM/DD/YYYY)                            Flight Number 
                                       0                                        0                                        0 
                             Tail Number                      Destination Airport                 Scheduled departure time 
                                     200                                        0                                        0 
                   Actual departure time         Scheduled elapsed time (Minutes)            Actual elapsed time (Minutes) 
                                       0                                        0                                        0 
               Departure delay (Minutes)                          Wheels-off time                  Taxi-Out time (Minutes) 
                                       0                                        0                                        0 
                 Delay Carrier (Minutes)                  Delay Weather (Minutes) Delay National Aviation System (Minutes) 
                                       0                                        0                                        0 
                Delay Security (Minutes)    Delay Late Aircraft Arrival (Minutes) 
                                       0                                        0 

As we see above that Tail Number has 200 empty cells.

Now I am going to see the rows of that particular column to find the reason of empty cells

#This command is use to see all the empty rows.
Total_data[`Tail Number`==""]

As we see in all these rows scheduled departure time was given but Actual departure time 00:00 ,Actual elapsed time is also 0 and all sort of delay columns are also 0 ,so by this we can conclude that the flights has got cancled for some reason or the data has not been captured for this particular flights.So this kind of data is not much used in the prediction so for now we are removing all the rows which are empty.

Removing all the rows which are empty.

setkey(Total_data,`Tail Number`)
Total_data<-Total_data[!""]

Now again checking wheather all the empty cell has got removed or not in train data

#Checking the NA 'S at entire data set of train and test
cat("The Number of blank cells  at Training data set: ", sum(Total_data==""))

The Number of blank cells  at Training data set:  0

so Now we our training data set free of empty cells.

Generally we should find and treate the outliers but due to less time issues and some Machine Learning Algorithm which i am using to build model are robust to outlier so for now i am not considering it.

Feature Engineering

Reading the Coordinates of the Destination Airports from the file.

lat_long<-as.data.table(read_excel("Lat_Long.xlsx"))
#converting string column as factor.
lat_long$Destination<-as.factor(lat_long$Destination)

Adding longitude and latitude columns for Origin Airport.

#Adding the columns
Total_data$`Start Station Longitude`<- -73.8630
Total_data$`Start Station Latitude`<- 40.7636

Adding Longitude and Latitude of Destination Airports.

Total_data<- merge(Total_data,lat_long,by.x ="Destination Airport",by.y = "Destination")

Adding the column of Geo_distance between source airport to destination airport.

Total_data[,distance := round(distHaversine(matrix(c(`Start Station Longitude`,`Start Station Latitude`), ncol = 2),matrix(c(Long,Lat),ncol = 2),r=6378137),2)]

we are removing these columns because these columns does not effect our prediction. 1) Flight Number 2) Destination Airport 3) Carrier Code 4) Tail Number 5) Start Station Longitude 6) Start Station Latitude 7) Lat 8) Long

Total_data[ ,`:=`(`Flight Number` = NULL, `Destination Airport` = NULL,`Carrier Code`= NULL,`Tail Number`= NULL,`Start Station Longitude`=NULL,`Start Station Latitude`=NULL,Lat=NULL,Long=NULL)]

so now we are converting the date columns to the date formate.

#Now converting the date column.
#we found 2 date format in data
mdy <- mdy(Total_data$`Date (MM/DD/YYYY)`)
dmy <- dmy(Total_data$`Date (MM/DD/YYYY)`)

 207846 failed to parse.

mdy[is.na(mdy)] <- dmy[is.na(mdy)] # some dates are ambiguous, here we give
Total_data$`Date (MM/DD/YYYY)`<- mdy # mdy precedence over dmy

Now we are dividing the following columns by hours and minutes 1)1)Scheduled departure time 2)Actual departure time

#Now we are dividing Scheduled departure time column to scheduled_departure_time_hours and scheduled_departure_time_minutes
Total_data$`Scheduled departure time hours`<-hour(strptime(Total_data$`Scheduled departure time`, format = '%H:%M', 'GMT'))
Total_data$`Scheduled departure time minutes`<-minute(strptime(Total_data$`Scheduled departure time`,format = '%H:%M', 'GMT'))
#Now we are dividing Actual departure time departure time column to Actual_departure_time_hours and Actual_departure_time_minutes
Total_data$`Actual departure time hours`<-hour(strptime(Total_data$`Actual departure time`, format = '%H:%M', 'GMT'))
Total_data$`Actual departure time minutes`<-minute(strptime(Total_data$`Actual departure time`,format = '%H:%M', 'GMT'))
#Now removing the original Scheduled departure time column
Total_data$`Scheduled departure time`<-NULL
#Now removing the actual departure time column
Total_data$`Actual departure time`<-NULL
#Now removing the wheels of time as its not useful at model building.
Total_data$`Wheels-off time`<-NULL

Now extracting months and weekdays from the Date column in train data set.Removing Date column

#this command is use to extract months
Total_data$months<-as.factor(month(Total_data$`Date (MM/DD/YYYY)`))
#this command is used to extract weekdays
Total_data$weekDays<-as.factor(weekdays(Total_data$`Date (MM/DD/YYYY)`))
#Removing Date column
#train$`Date (MM/DD/YYYY)`<-NULL

Now extracting seasons from Date in the training data set.we are extracting seasons according to USA. 1) Summer - June,July,August (6,7,8) 2) Autumn - Sept,Oct,Nov (9,10,11) 3) Winter - December,Jan,Feb (12,1,2) 4) Spring - March,Aprial,May (3,4,5) Now creating the new column of seasons.

indx <- setNames( rep(c('winter', 'spring', 'summer',
                   'fall'),each=3), c(c(12,1,2),c(3,4,5),c(6,7,8),c(9,10,11)))
Total_data[,seasons := as.factor(unname(indx[as.character(month(Total_data$`Date (MM/DD/YYYY)`))]))]

Now extracting years from the date.

Total_data$years<-as.factor(year(Total_data$`Date (MM/DD/YYYY)`))

2016

Now we are subsetting data of 2016.

train_2016<-Total_data[years=="2016"]
#extracting day 
train_2016$day<-as.factor(day(train_2016$`Date (MM/DD/YYYY)`))

Holidays list of USA at Year 2016.

Holidays

Day Dates

Fri Jan 1 Mon Jan 18 Mon Feb 15 Fri Apr 15 Sun May 8 Mon May 30 Sun June 19 Mon July 4 Mon Sept 5 Mon Oct 10 Fri Nov 11 Thur Nov 24 Fri Nov 25 Mon Dec 26

Now we are creating the column Holiday for the train_2016 data.

#Inserting the holiday column. 
train_2016$Holiday<-as.factor(ifelse((train_2016$months == "1" & train_2016$weekDays=="Friday" & train_2016$day=="1") | (train_2016$months == "1" & train_2016$weekDays=="Monday" & train_2016$day=="18") | (train_2016$months == "2" & train_2016$weekDays=="Monday" & train_2016$day=="15") | (train_2016$months == "4" & train_2016$weekDays=="Friday" & train_2016$day=="15") |(train_2016$months == "5" & train_2016$weekDays=="Sunday" & train_2016$day=="8") |(train_2016$months == "5" & train_2016$weekDays=="Monday" & train_2016$day=="30") |(train_2016$months == "6" & train_2016$weekDays=="Sunday" & train_2016$day=="19") |(train_2016$months == "7" & train_2016$weekDays=="Monday" & train_2016$day=="4") |(train_2016$months == "9" & train_2016$weekDays=="Monday" & train_2016$day=="5") |(train_2016$months == "10" & train_2016$weekDays=="Monday" & train_2016$day=="10") |(train_2016$months == "11" & train_2016$weekDays=="Friday" & train_2016$day=="11") |(train_2016$months == "11" & train_2016$weekDays=="Thursday" & train_2016$day=="24") |(train_2016$months == "11" & train_2016$weekDays=="Friday" & train_2016$day=="25") |(train_2016$months == "12" & train_2016$weekDays=="Monday" & train_2016$day=="26")== TRUE,1,0))

1 - means yes there was holiday.

0 - means No there was no holiday.

Holidays list of USA at Year 2017.

2017

Now subsetting the 2017 data from train.

train_2017<-Total_data[years=="2017"]
#extracting day 
train_2017$day<-as.factor(day(train_2017$`Date (MM/DD/YYYY)`))

Holidays list of USA at Year 2017.

Holidays

Day Dates

Monday Jan 2 Monday Jan 16 Monday Feb 20 Monday April 17 Sunday May 14 Monday May 29 Sunday June 18 Tuesday July 4 Monday September 04 Monday october 09 Friday November 10 Saturday November 11 Monday November 13 Thursday November 23 Friday November 24 Monday December 25

Now we are creating the column Holiday for the train_2016 data.

#Inserting the holiday column. 
train_2017$Holiday <- as.factor(ifelse((train_2017$months == "1" & train_2017$weekDays=="Monday" & train_2017$day=="2") | (train_2017$months == "1" & train_2017$weekDays=="Monday" & train_2017$day=="16") | (train_2017$months == "2" & train_2017$weekDays=="Monday" & train_2017$day=="20") | (train_2017$months == "4" & train_2017$weekDays=="Monday" & train_2017$day=="17") |(train_2017$months == "5" & train_2017$weekDays=="Sunday" & train_2017$day=="14") |(train_2017$months == "5" & train_2017$weekDays=="Monday" & train_2017$day=="29") |(train_2017$months == "6" & train_2017$weekDays=="Sunday" & train_2017$day=="18") |(train_2017$months == "7" & train_2017$weekDays=="Tuesday" & train_2017$day=="4") |(train_2017$months == "9" & train_2017$weekDays=="Monday" & train_2017$day=="4") |(train_2017$months == "10" & train_2017$weekDays=="Monday" & train_2017$day=="9") |(train_2017$months == "11" & train_2017$weekDays=="Friday" & train_2017$day=="10") |(train_2017$months == "11" & train_2017$weekDays=="Saturday" & train_2017$day=="11") |(train_2017$months == "11" & train_2017$weekDays=="Monday" & train_2017$day=="13") |(train_2017$months == "11" & train_2017$weekDays=="Thursday" & train_2017$day=="23")|(train_2017$months == "11" & train_2017$weekDays=="Friday" & train_2017$day=="24")|(train_2017$months == "12" & train_2017$weekDays=="Monday" & train_2017$day=="25") == TRUE,1,0))

Now joining the Two years data set.

Total_data1<-rbind(setDT(train_2016), setDT(train_2017), fill=TRUE)
#removing unwanted data set
rm(train_2016)
rm(train_2017)

2018

test<-Total_data[years=="2018"]
test$day<-day(test$`Date (MM/DD/YYYY)`)

Holidays list of USA at Year 2018.

Holidays

Day Dates

Monday Jan 1 Monday Jan 15 Monday Feb 19 Saturday March 31 Monday May 28 Wednesday July 4 Monday Sept 3 Monday Nov 12 Thursday Nov 22 Friday Nov 23 Tuesday December 25

Now we are creating the column Holiday for the train_2016 data.

#Inserting the holiday column. 
test$Holiday<-as.factor(ifelse((test$months == "1" & test$weekDays=="Monday" & test$day=="1") | (test$months == "1" & test$weekDays=="Monday" & test$day=="15") | (test$months == "2" & test$weekDays=="Monday" & test$day=="19") | (test$months == "3" & test$weekDays=="Saturday" & test$day=="31") |(test$months == "5" & test$weekDays=="Monday" & test$day=="28") |(test$months == "7" & test$weekDays=="Wednesday" & test$day=="4") |(test$months == "9" & test$weekDays=="Monday" & test$day=="3") |(test$months == "11" & test$weekDays=="Monday" & test$day=="12") |(test$months == "11" & test$weekDays=="Thursday" & test$day=="22") |(test$months == "11" & test$weekDays=="Friday" & test$day=="23") |(test$months == "12" & test$weekDays=="Tuesday" & test$day=="25")== TRUE,1,0))

Joining the tables.

Total_data1<-rbind(setDT(Total_data1), setDT(test), fill=TRUE)

2019

test<-Total_data[years=="2019"]
test$day<-day(test$`Date (MM/DD/YYYY)`)

Holidays list of USA at Year 2019.

Holidays

Reason Day Date

New Year’s Day Monday January 1

Birthday of Martin Luther King Jr.: Monday January 21

Birthday of George Washington (President’s Day): Monday February 18

Memorial Day: Monday May 27

Independence Day: Thursday July 4

Labor Day: Monday September 2

Columbus Day: Monday October 14

Veterans Day: Monday November 11

Thanksgiving Day: Thursday November 28

Christmas Day: Wednesday December 25

Other important holidays to note that you don’t get off for:

Groundhog Day: Saturday February 2

Valentine’s Day: Thursday February 14

Saint Patrick’s Day: Sunday March 17

Tax Day: Monday April 15

Good Friday: Friday April 19

Easter: Sunday April 21

Cinco de Mayo: Sunday May 5

Mother’s Day: Sunday May 12

Father’s Day: Sunday June 16

Halloween: Thursday October 31

Black Friday: Friday November 29

#Inserting the holiday column. 
test$Holiday<-as.factor(ifelse((test$months == "1" & test$weekDays=="Monday" & test$day=="1") | (test$months == "1" & test$weekDays=="Monday" & test$day=="21") | (test$months == "2" & test$weekDays=="Monday" & test$day=="18") | (test$months == "5" & test$weekDays=="Monday" & test$day=="27") |(test$months == "7" & test$weekDays=="Thursday" & test$day=="4") |(test$months == "9" & test$weekDays=="Monday" & test$day=="2") |(test$months == "10" & test$weekDays=="Monday" & test$day=="14") |(test$months == "11" & test$weekDays=="Monday" & test$day=="11") |(test$months == "11" & test$weekDays=="Thursday" & test$day=="28") |(test$months == "12" & test$weekDays=="Wednesday" & test$day=="25") |(test$months == "2" & test$weekDays=="Saturday" & test$day=="2")|(test$months == "2" & test$weekDays=="Thursday" & test$day=="14")|(test$months == "3" & test$weekDays=="Sunday" & test$day=="17")|(test$months == "4" & test$weekDays=="Monday" & test$day=="15")|(test$months == "4" & test$weekDays=="Friday" & test$day=="19")|(test$months == "4" & test$weekDays=="Sunday" & test$day=="21")|(test$months == "5" & test$weekDays=="Sunday" & test$day=="5")|(test$months == "5" & test$weekDays=="Sunday" & test$day=="12")|(test$months == "6" & test$weekDays=="Sunday" & test$day=="16")|(test$months == "10" & test$weekDays=="Thursday" & test$day=="31")|(test$months == "11" & test$weekDays=="Friday" & test$day=="29")== TRUE,1,0))

Joining the tables.

Total_data<-rbind(setDT(Total_data1), setDT(test), fill=TRUE)
rm(Total_data1)

Now extracting Morning,Noon,Evening and Night,the following the timing classifications. 5 Hours - 11 Hours :- Morning 12 Hours - 16 Hours :- Noon 17 Hours - 19 Hours :- Evening 20 Hours - 4 Hours :- Night.

Now inserting th DayCycle column into train data set.

Total_data[,DayCycle:=as.factor( ifelse(
  `Scheduled departure time hours` >= 5 & `Scheduled departure time hours`<=11 , 'Morning',
    ifelse(`Scheduled departure time hours` >= 12 & `Scheduled departure time hours`<=16,'Noon' ,
      ifelse(`Scheduled departure time hours` >= 17 & `Scheduled departure time hours`<=19, 'Evening', 
          "Night"   )
    )
  )
)
]

creating the target variable by the formula ((actual_elapsed time - scheduled_elapsed time)+departure time)<=0 no_delay >=1 is delay

#creating the target variable which is categorical variable.
Total_data$Delay<-as.factor(ifelse((Total_data$`Actual elapsed time (Minutes)`- Total_data$`Scheduled elapsed time (Minutes)`)+Total_data$`Departure delay (Minutes)` <= 0,0,1))

1. 1 - Delayed
2. 0 - Not Delayed

Now removing unwanted features from the train data set.

Total_2018<-Total_data
Total_2018$years<-NULL
Total_2018$day<-NULL
Total_2018$`Date (MM/DD/YYYY)`<-NULL
rm(test)

Removing the actual elaps time ,Scheduled elapsed time (Minutes),Departure delay (Minutes) from trainig and testing data as these three columns are highly correlated to target varible.

#Removing from traing data
Total_2018$`Actual elapsed time (Minutes)`<-NULL
Total_2018$`Departure delay (Minutes)`<-NULL
Total_2018$`Scheduled elapsed time (Minutes)`<-NULL

#Parameter for the function.
#parameter 1 :- Number of correct predictions for class 0
#parameter 2 :- Number of wrong predictions for class 0
#parameter 3 :- Number of correct predictions for class 1
#parameter 4 :- Number of wrong predictions for class 1
graphConfusionMatrix<-function(correct_class_a,wrong_class_a,correct_class_b,wrong_class_b){
  
  #Creating the data for the plot.
  x <- data.frame("About_results" = c("Correct_Prediction","Wrong_Prediction"), "Values" = c(correct_class_a,wrong_class_a))
  y <- data.frame("About_results" = c("Correct_Prediction","Wrong_Prediction"), "Values" = c(correct_class_b,wrong_class_b))
  
  #ploting the graph.
  p <- plot_ly() %>%
   add_pie(data = count(x,About_results), labels = ~About_results, values =x$Values ,
         name = "class 0", domain = list(row = 0, column = 0)) %>%
 add_pie(data = count(y,About_results), labels = ~About_results, values = y$Values,
         name = "class 1", domain = list(row = 0, column = 1)) %>%
   layout(title = "Visualization Of Confusion Matrix",showlegend = T,
        grid=list(rows=1, columns=2),
        xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),
        yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))
  #Displaying the graphs
  p
  #removing unwanted dataframes.
  #rm(x)
  #rm(y)
}

names(Total_2018) <- make.names(names(Total_2018))

Encoding the data for XGBoost Model.

#Encoding the  data set
Total_data_encoded<-one_hot(Total_2018)

While doing onehot encoding we have got two columns for the target variable.so we should remove one of the column

Total_data_encoded$Delay_0<-NULL

dtestfinal<-xgb.DMatrix(as.matrix(Total_data_encoded[,-"Delay_1"]), label = as.matrix(Total_data_encoded$Delay_1))

Now importing out model.

model<- xgb.load('xgb_all_data.model')

Now we are going to fit the new data to our Model.

xgbpred <- predict (model,dtestfinal,type = "class")

As our model threshold is 0.5, now we are classifing the predictions.

xgbpred <- ifelse (xgbpred >= 0.5,1,0)

Now we are going to check on confusion matrix

confusionMatrix(as.factor(Total_data_encoded$Delay_1),as.factor(xgbpred),positive = "1")

Confusion Matrix and Statistics

          Reference
Prediction      0      1
         0 200490  21952
         1  27903  91499
                                         
               Accuracy : 0.8542         
                 95% CI : (0.853, 0.8553)
    No Information Rate : 0.6681         
    P-Value [Acc > NIR] : < 2.2e-16      
                                         
                  Kappa : 0.6754         
                                         
 Mcnemar's Test P-Value : < 2.2e-16      
                                         
            Sensitivity : 0.8065         
            Specificity : 0.8778         
         Pos Pred Value : 0.7663         
         Neg Pred Value : 0.9013         
             Prevalence : 0.3319         
         Detection Rate : 0.2677         
   Detection Prevalence : 0.3493         
      Balanced Accuracy : 0.8422         
                                         
       'Positive' Class : 1              
                                         

graphConfusionMatrix(200490 ,21952, 91499,27903)

Lets see which is the best threshold value for our model.

Now Drawing the ROC cURVE for finding out the ideal threshold value at which true positive and false positive get balanced.

ROCpred<-prediction(xgbpred,Total_data_encoded$Delay_1)
ROCperf<-performance(ROCpred,"tpr","fpr")
plot(ROCperf,col="blue",print.cutoffs.at=seq(0.1,by=0.1,text.adj=c(-0.2,1.7),cex=0.7))

In seq.default(0.1, by = 0.1, text.adj = c(-0.2, 1.7), cex = 0.7) :
 extra arguments 㤼㸱text.adj㤼㸲, 㤼㸱cex㤼㸲 will be disregarded

yes its predicting good for the test data.

Building the model on this data.

#default parameters
#params <- list(booster = "gbtree", objective = "binary:logistic", eta=0.2, max_depth=4, min_child_weight=2, subsample=1, colsample_bytree=1)

#find best nround
#cv<-xgb.cv( params = params, data = dtestfinal, nrounds = 1000, nfold = 5,gamma=0, showsd = T, stratified = T, print.every.n = 10, early.stop.round = 20, maximize = F)
#cv$best_iteration
#print(cv)
#print(cv, verbose=TRUE)

#first default - model training
#xgb1 <- xgb.train (params = params, data = dtestfinal, nrounds = 973, watchlist = list(train=dtestfinal), print.every.n = 10, early.stop.round = 10, maximize = F , eval_metric = "error")